Multi-armed Bandits with Metric Switching Costs

In this paper we consider the stochastic multi-armed bandit with metric switching costs. Given a set of locations (arms) in a metric space and prior information about the reward available at these locations, cost of getting a sample/play at every location and rules to update the prior based on samples/plays, the task is to maximize a certain objective function constrained to a distance cost of L and cost of plays C. This fundamental problem models several stochastic optimization problems in robot navigation, sensor networks, labor economics, etc. In this paper we consider two natural objective functions – future utilization and past utilization. We develop a common duality-based framework to provide the first O(1) approximation in the metric switching cost model; the actual constants being quite small. Since both problems are Max-SNP hard, this result is the best possible. We also show an “adaptivity” result, namely, there exists a policy which orders the arms and visits them in that fixed order without revisiting any arm and this policy gives at least Ω(1) fraction reward of the fully adaptive policy. The overall technique and the ensuing structural results are independently of interest in the context of bandit problems with complicated side-constraints. As a side-effect, our techniques also improve the approximation ratio of the budgeted learning problem from 4 to 3 + . ∗Department of Computer and Information Sciences, University of Pennsylvania, Philadelphia PA 19104-6389. Email: [email protected]. Research supported in part by an Alfred P. Sloan Research Fellowship, and an NSF CAREER Award CCF-0644119. †Department of Computer Science, Duke University, Durham NC 27708-0129. Email: [email protected]. Research supported by NSF via a CAREER award and grant CNS-0540347.